Content recommendations based on organic keyword analysis

ABSTRACT

Systems, methods, and computer-readable storage media that may be used to generate recommendations based on organic search term analysis are provided. One method includes determining conversion path data for a content provider. The method further includes determining a plurality of organic search keywords within the conversion path data. The method further includes analyzing the plurality of organic search keywords within the conversion path data to generate an analysis metric for each of the plurality of organic search keywords. The method further includes selecting one or more of the plurality of organic search keywords based on the analysis metrics for the organic search keywords, and generating one or more recommendations for new content to be published by the content provider based on the selected one or more organic search keywords.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/085,499, filed Nov. 20, 2013, and is also a continuation of U.S.patent application Ser. No. 14/085,477 filed Nov. 20, 2013, both ofwhich are incorporated by reference herein in their entireties.

BACKGROUND

Content providers often publish content items in networked resourcesthrough online content management systems with the goal of having an enduser interact with (e.g., click through) the content items and purchasea product or service offered by the content providers. Content providerstraditionally have assigned credit for a conversion largely, if notentirely, to the last click prior to the conversion. Attribution isbased on the principle that conversion decisions are the cumulativeresult of many interactions (e.g., clicks, impressions, video views,etc.) over time, and not just the last click prior to a conversion.Evaluating keywords based solely on the last click prior to conversionignores the contributions made by keywords earlier in the conversionpaths.

SUMMARY

One illustrative implementation of the disclosure relates to a methodthat includes determining, at a computerized analysis system, conversionpath data for a content provider. The conversion path data includes datarelating to a plurality of conversion paths associated with the contentprovider leading to a plurality of conversions. Each of the plurality ofconversion paths includes one or more user actions leading to one of theplurality of conversions. The method further includes determining, atthe analysis system, a plurality of organic search keywords within theconversion path data. The plurality of organic search keywords arekeywords that are not included within a set of paid keywords associatedwith bids submitted by the content provider. The method further includesanalyzing, at the analysis system, the plurality of organic searchkeywords within the conversion path data to generate an analysis metricfor each of the plurality of organic search keywords. The method furtherincludes generating one or more recommendations for organic searchkeywords to add to the set of paid keywords based on the analysismetrics for the plurality of search keywords.

Another implementation relates to a system that includes at least onecomputing device operably coupled to at least one memory and configuredto determine conversion path data for a content provider. The conversionpath data includes data relating to a plurality of conversion pathsassociated with the content provider leading to a plurality ofconversions. Each of the plurality of conversion paths includes one ormore user actions leading to one of the plurality of conversions. The atleast one computing device is further configured to determine aplurality of organic search keywords within the conversion path data.The plurality of organic search keywords are keywords that are notincluded within a set of paid keywords associated with bids submitted bythe content provider. The at least one computing device is furtherconfigured to analyze the plurality of organic search keywords withinthe conversion path data to generate an analysis metric for each of theplurality of organic search keywords. The at least one computing deviceis further configured to generate one or more recommendations fororganic search keywords to add to the set of paid keywords based on theanalysis metrics for the plurality of search keywords.

Yet another implementation relates to one or more computer-readablestorage media having instructions stored thereon that, when executed byat least one processor, cause the at least one processor to performoperations. The operations include determining conversion path data fora content provider. The conversion path data includes data relating to aplurality of conversion paths associated with the content providerleading to a plurality of conversions. Each of the plurality ofconversion paths includes one or more user actions leading to one of theplurality of conversions. The operations further include determining aplurality of organic search keywords within the conversion path data.The plurality of organic search keywords are keywords that are notincluded within a set of paid keywords associated with bids submitted bythe content provider. The operations further include analyzing theplurality of organic search keywords within the conversion path data togenerate an analysis metric for each of the plurality of organic searchkeywords. The analysis metric includes a conversion contribution metricrelated to how directly the organic search keyword contributes to theplurality of conversions. The operations further include generating oneor more recommendations for organic search keywords to add to the set ofpaid keywords based on the analysis metrics for the plurality of searchkeywords. The operations further include providing the one or morerecommendations to the content provider within an interface throughwhich the content provider can select one or more of the recommendationsfor implementation within the set of paid keywords.

Another illustrative implementation of the disclosure relates to amethod that includes determining, at a computerized analysis system,conversion path data for a content provider. The conversion path dataincludes data relating to a plurality of conversion paths associatedwith the content provider leading to a plurality of conversions, andeach of the plurality of conversion paths includes one or more useractions leading to one of the plurality of conversions. The methodfurther includes determining, at the analysis system, a plurality oforganic search keywords within the conversion path data. The pluralityof organic search keywords are keywords that are not included within aset of paid keywords associated with bids submitted by the contentprovider. The method further includes analyzing, at the analysis system,the plurality of organic search keywords within the conversion path datato generate an analysis metric for each of the plurality of organicsearch keywords. The method further includes selecting one or more ofthe plurality of organic search keywords based on the analysis metricsfor the organic search keywords, and generating one or morerecommendations for new content to be published by the content providerbased on the selected one or more organic search keywords.

Another implementation relates to a system including at least onecomputing device operably coupled to at least one memory. The at leastone computing device is configured to determine conversion path data fora content provider. The conversion path data includes data relating to aplurality of conversion paths associated with the content providerleading to a plurality of conversions. Each of the plurality ofconversion paths includes one or more user actions leading to one of theplurality of conversions. The at least one computing device is furtherconfigured to determine a plurality of organic search keywords withinthe conversion path data. The plurality of organic search keywords arekeywords that are not included within a set of paid keywords associatedwith bids submitted by the content provider. The at least one computingdevice is further configured to analyze the plurality of organic searchkeywords within the conversion path data to generate an analysis metricfor each of the plurality of organic search keywords. The at least onecomputing device is further configured to select one or more of theplurality of organic search keywords based on the analysis metrics forthe organic search keywords, and to generate one or more recommendationsfor new content to be published by the content provider based on theselected one or more organic search keywords.

Yet another implementation relates to one or more computer-readablestorage media having instructions stored thereon that, when executed byat least one processor, cause the at least one processor to performoperations. The operations include determining conversion path data fora content provider. The conversion path data includes data relating to aplurality of conversion paths associated with the content providerleading to a plurality of conversions. Each of the plurality ofconversion paths includes one or more user actions leading to one of theplurality of conversions. The operations further include determining aplurality of organic search keywords within the conversion path data.The plurality of organic search keywords are keywords that are notincluded within a set of paid keywords associated with bids submitted bythe content provider. The operations further include analyzing theplurality of organic search keywords within the conversion path data togenerate an analysis metric for each of the plurality of organic searchkeywords. The analysis metric includes a conversion contribution metricrelated to how directly the organic search keyword contributes to theplurality of conversions. The operations further include selecting oneor more of the plurality of organic search keywords based on theanalysis metrics for the organic search keywords. The operations furtherinclude generating one or more recommendations for new content to bepublished by the content provider based on the selected one or moreorganic search keywords, and providing the one or more recommendationsto the content provider.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other features, aspects, andadvantages of the subject matter will become apparent from thedescription, the drawings, and the claims.

FIG. 1 is a block diagram of an analysis system and associatedenvironment according to an illustrative implementation.

FIG. 2 is a flow diagram of a process for generating recommendations forpaid keywords based on analysis of organic search keywords in conversionpath data according to an illustrative implementation.

FIG. 3A is a visual representation of conversion path data according toan illustrative implementation.

FIG. 3B is an illustration of a user interface configured to provide thepaid keyword recommendations generated using the process of FIG. 2 to acontent provider according to an illustrative implementation.

FIG. 4 is a flow diagram of a process for generating recommendations fortypes of new content a content publisher may want to consider publishingbased on analysis of organic search keywords in conversion path dataaccording to an illustrative implementation.

FIG. 5 is an illustration of a user interface configured to provide thecontent recommendations generated using the process of FIG. 4 to acontent provider according to an illustrative implementation.

FIG. 6 is a block diagram of a computing system according to anillustrative implementation.

DETAILED DESCRIPTION

Referring generally to the Figures, various illustrative systems andmethods are provided that may be used to generate recommendations tocontent providers based on analysis of organic search keywords inconversion path data. Many content providers look primarily at a singleuser activity to determine whether a displayed content item issuccessful (e.g., whether the user converted or did not convert),without considering the entire path of user activity that leads toconversions. On this basis, content providers may focus heavily onlast-click keywords, or keywords associated with the final user activityor selection prior to a conversion, when deciding paid keywords on whichto bid in an auction system for displaying content items. This focusdoes not account for the contributions that user activity earlier in theconversion paths make towards ultimately driving the conversions.

In order to evaluate keywords in a manner that does not purely examinethe last click, a content provider may look at a limited amount (e.g.,18 months) of historical conversion path data to understand how paidkeywords have performed across the marketing funnel. However, theresulting models are limited by the fact that the only keywords investedin historically were those that performed favorably in the traditionallast-click measurement system. Any truly upper-funnel keywords, orkeywords that appear early in the conversion paths, would have beenturned off early and not been invested in due to perception of poorperformance, and would not be identified in such a resulting model.

This disclosure provides systems and methods for examining organicsearch keywords in a content provider's conversion paths and generatingnew keyword recommendations based on the analysis. An illustrativesystem may receive or generate a set of organic search keyword datausing conversion path data associated with a content provider. In someimplementations, the organic search keyword data may be generated byremoving paid keywords from a set of conversion path data including bothpaid and organic path data, leaving only the organic search keywords ina resulting organic conversion path data set.

The resulting organic search keyword data may be analyzed to generateone or more new keyword recommendations. In some implementations, someor all of the organic search keywords included in the organic searchkeyword data may be evaluated for their contribution to conversions. Forexample, in some implementations, an assist-to-last ratio may becalculated for each organic keyword, as described in further detailbelow. The assist-to-last ratio and/or another conversion contributionmeasure may be used to determine a relative position in the conversionfunnel for each organic keyword. Upper-funnel keywords may representhigher opportunity as well as higher risk; such keywords may beassociated with longer conversion times and more compoundinginteractions to reach a conversion, but also are more likely to beignored by other competitors in an auction and be less expensive to thecontent provider than lower-funnel keywords. Lower-funnel keywords, orkeywords that appear close to conversions in the conversion paths, mayhave shorter conversion times and be more directly correlated toconversions, but may be heavily focused upon by other competitors andmore expensive.

In some implementations, factors other than conversion contribution maybe considered when generating and/or presenting recommendations. Forexample, a cost metric (e.g., an anticipated cost per click, or CPC) maybe generated for one or more of the analyzed organic keywords indicatingan estimated cost to the content provider if the content provider addedthe organic keyword as a paid keyword. In some implementations, anormalized value may be used against other paid keyword portfolio termsto highlight terms with greatest potential (e.g., a relative costindication rather than an absolute cost to the content provider). Insome implementations, a new visitor metric may be generated to placegreater importance or weight on those organic keywords that tend todrive new traffic to the content provider versus driving returningcustomers to convert.

The system may generate a report providing one or more new keywordrecommendations based on the analyzed organic keywords. In someimplementations, the analyzed organic keyword opportunities may becompared against the content provider's current portfolio of paidkeywords to separate new opportunities and/or to provide a comparativeevaluation for the content provider against those terms for which thecontent provider has previously submitted paid bids. The recommendationsmay be provided through an interface which allows the content providerto accept and/or reject the recommendations. In some implementations,the accepted recommendations may automatically be processed and added tothe content provider's portfolio of paid keywords. In someimplementations, the reactions (e.g., acceptances or rejections) of thecontent provider to the recommendations may be used in determiningsubsequent keyword recommendations.

In some implementations, the analyzed organic keywords may additionallyor alternatively be used to generate recommendations for content thecontent provider may wish to consider providing to users, such as newwebpages, mobile applications, etc. that may be of interest to usersbased on the organic search keyword analysis. For example, the organicsearch keyword analysis may indicate that a frequent upper-funnel phrasein the conversion paths for a fictional Acme Shoe Company is “marathontraining.” In such an implementation, a recommendation may be providedto the content provider to publish a section on its website dedicated toinformational resources on training for a marathon. Such resources maygenerate upper-funnel traffic to Acme Shoe Company's website and placeAcme in the forefront of the consumer's mind early in thedecision-making process, which may help generate more conversions.

In some implementations, organic search keyword analysis may be used toevaluate traffic navigating to existing content, or to the new contentafter it is generated. For example, the organic search keyword analysismay be used to assess the contribution of the traffic to the contenttowards eventually converting (e.g., using assist-to-last ratios) andmake recommendations for building similar content and/or for using paidsearch to scale the volume of traffic sent to the content.

In some implementations, organic search keyword data may be generatedand collected by systems operated by an operator of the analysis system,and/or recommendations may be made to modify paid keywords used inbidding on content items to be published by a content management systemoperated by the operator of the analysis system. In someimplementations, some or all of the organic search keyword data may bereceived from third parties, such as content distribution networks forwhich keywords associated with published content with which usersinteract is at least partially available to the operator of the analysissystem. In some implementations, recommendations may be made that may beused to modify a set of paid keywords used in making bids to publishcontent using a third-party content management system.

For situations in which the systems discussed here collect personalinformation about users, or may make use of personal information, theusers may be provided with an opportunity to control whether programs orfeatures that may collect personal information (e.g., information abouta user's social network, social actions or activities, a user'spreferences, or a user's current location), or to control whether and/orhow to receive content from the content server that may be more relevantto the user. In addition, certain data may be anonymized in one or moreways before it is stored or used, so that personally identifiableinformation is removed when generating parameters (e.g., demographicparameters). For example, a user's identity may be anonymized so that nopersonally identifiable information can be determined for the user, or auser's geographic location may be generalized where location informationis obtained (such as to a city, ZIP code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about him or her and usedby a content server.

Referring now to FIG. 1, and in brief overview, a block diagram of ananalysis system 150 and associated environment 100 is shown according toan illustrative implementation. One or more user devices 104 may be usedby a user to perform various actions and/or access various types ofcontent, some of which may be provided over a network 102 (e.g., theInternet, LAN, WAN, etc.). For example, user devices 104 may be used toaccess websites (e.g., using an internet browser), media files, and/orany other types of content. A content management system 108 may beconfigured to select content for display to users within resources(e.g., webpages, applications, etc.) and to provide content items 112from a content database 110 to user devices 104 over network 102 fordisplay within the resources. The content from which content managementsystem 108 selects items may be provided by one or more contentproviders via network 102 using one or more content provider devices106.

In some implementations, bids for content to be selected by contentmanagement system 108 may be provided to content management system 108from content providers participating in an auction using devices, suchas content provider devices 106, configured to communicate with contentmanagement system 108 through network 102. In such implementations,content management system 108 may determine content to be published inone or more content interfaces of resources (e.g., webpages,applications, etc.) shown on user devices 104 based at least in part onthe bids.

In some implementations, an analysis system 150 may be configured toanalyze organic search keyword data relating to conversion paths for acontent provider and develop recommendations for new paid keywords thecontent provider may wish to bid on in auctions and/or new content thecontent provider may wish to consider publishing. Organic searchkeywords are keywords that are not a part of the set of paid keywords114 associated with auction bids submitted by the content provider.Organic search keywords may be keywords included in search queriesleading to a user interaction (e.g., click/selection, being presentedwith an impression, etc.) with a content item that is not shown as theresult of a paid bid by the content provider and/or other contentproviders. The conversion path data may include multiple conversionpaths, and each conversion path may include one or more user actionsthat ultimately lead to a conversion (e.g., a sale of a product and/orservice). The conversion path data may include user actions associatedwith organic search keywords as well as user actions associated withpaid keywords.

Analysis system 150 may be configured to analyze the organic searchkeyword data to generate one or more analysis metrics. For example,analysis system 150 may generate a metric for each organic searchkeyword representing whether the organic search keyword appeared mostfrequently near the top of the marketing funnel (e.g., in connectionwith early user interactions, further away from the end conversions) ornear the bottom of the funnel (e.g., in connection with later userinteractions, near the end conversions). Analysis system 150 may beconfigured to utilize the metric to generate one or morerecommendations. In some implementations, analysis system 150 maysuggest adding one or more of the organic search keywords to the contentprovider's paid keyword portfolio. For example, adding upper-funnelorganic search keywords may help increase awareness of the contentprovider's brands. In some implementations, analysis system 150 maysuggest new types of content the content provider might considercreating and publishing based on the organic search keywords. Forexample, if a particular organic search keyword appears frequently nearthe top of the marketing funnel in the content provider's conversionpath data, developing content directed to the subject matter of thatorganic search keyword may help raise awareness of the contentprovider's brands early in users' search processes, and may ultimatelyhelp drive a larger number of conversions for the content provider.

Referring still to FIG. 1, and in greater detail, user devices 104and/or content provider devices 106 may be any type of computing device(e.g., having a processor and memory or other type of computer-readablestorage medium), such as a television and/or set-top box, mobilecommunication device (e.g., cellular telephone, smartphone, etc.),computer and/or media device (desktop computer, laptop or notebookcomputer, netbook computer, tablet device, gaming system, etc.), or anyother type of computing device. In some implementations, one or moreuser devices 104 may be set-top boxes or other devices for use with atelevision set. In some implementations, content may be provided via aweb-based application and/or an application resident on a user device104. In some implementations, user devices 104 and/or content providerdevices 106 may be designed to use various types of software and/oroperating systems. In various illustrative implementations, user devices104 and/or content provider devices 106 may be equipped with and/orassociated with one or more user input devices (e.g., keyboard, mouse,remote control, touchscreen, etc.) and/or one or more display devices(e.g., television, monitor, CRT, plasma, LCD, LED, touchscreen, etc.).

User devices 104 and/or content provider devices 106 may be configuredto receive data from various sources using a network 102. In someimplementations, network 102 may comprise a computing network (e.g.,LAN, WAN, Internet, etc.) to which user devices 104 and/or contentprovider device 106 may be connected via any type of network connection(e.g., wired, such as Ethernet, phone line, power line, etc., orwireless, such as WiFi, WiMAX, 3G, 4G, satellite, etc.). In someimplementations, network 102 may include a media distribution network,such as cable (e.g., coaxial metal cable), satellite, fiber optic, etc.,configured to distribute media programming and/or data content.

Content management system 108 may be configured to conduct a contentauction among third-party content providers to determine whichthird-party content is to be provided to a user device 104. For example,content management system 108 may conduct a real-time content auction inresponse to a user device 104 requesting first-party content from acontent source (e.g., a website, search engine provider, etc.) orexecuting a first-party application. Content management system 108 mayuse any number of factors to determine the winner of the auction. Forexample, the winner of a content auction may be based in part on thethird-party content provider's bid and/or a quality score for thethird-party provider's content (e.g., a measure of how likely the userof the user device 104 is to click on the content). In other words, thehighest bidder is not necessarily the winner of a content auctionconducted by content management system 108, in some implementations.

Content management system 108 may be configured to allow third-partycontent providers to create campaigns to control how and when theprovider participates in content auctions. A campaign may include anynumber of bid-related parameters, such as a minimum bid amount, amaximum bid amount, a target bid amount, or one or more budget amounts(e.g., a daily budget, a weekly budget, a total budget, etc.). In somecases, a bid amount may correspond to the amount the third-partyprovider is willing to pay in exchange for their content being presentedat user devices 104. In some implementations, the bid amount may be on acost per impression or cost per thousand impressions (CPM) basis. Infurther implementations, a bid amount may correspond to a specifiedaction being performed in response to the third-party content beingpresented at a user device 104. For example, a bid amount may be amonetary amount that the third-party content provider is willing to pay,should their content be clicked on at the client device, therebyredirecting the client device to the provider's webpage or anotherresource associated with the content provider. In other words, a bidamount may be a cost per click (CPC) bid amount. In another example, thebid amount may correspond to an action being performed on thethird-party provider's website, such as the user of the user device 104making a purchase. Such bids are typically referred to as being on acost per acquisition (CPA) or cost per conversion basis.

A campaign created via content management system 108 may also includeselection parameters that control when a bid is placed on behalf of athird-party content provider in a content auction. If the third-partycontent is to be presented in conjunction with search results from asearch engine, for example, the selection parameters may include one ormore sets of search keywords. For instance, the third-party contentprovider may only participate in content auctions in which a searchquery for “golf resorts in California” is sent to a search engine. Otherexample parameters that control when a bid is placed on behalf of athird-party content provider may include, but are not limited to, atopic identified using a device identifier's history data (e.g., basedon webpages visited by the device identifier), the topic of a webpage orother first-party content with which the third-party content is to bepresented, a geographic location of the client device that will bepresenting the content, or a geographic location specified as part of asearch query. In some cases, a selection parameter may designate aspecific webpage, website, or group of websites with which thethird-party content is to be presented. For example, an advertiserselling golf equipment may specify that they wish to place anadvertisement on the sports page of an particular online newspaper.

Content management system 108 may also be configured to suggest a bidamount to a third-party content provider when a campaign is created ormodified. In some implementations, the suggested bid amount may be basedon aggregate bid amounts from the third-party content provider's peers(e.g., other third-party content providers that use the same or similarselection parameters as part of their campaigns). For example, athird-party content provider that wishes to place an advertisement onthe sports page of an online newspaper may be shown an average bidamount used by other advertisers on the same page. The suggested bidamount may facilitate the creation of bid amounts across different typesof client devices, in some cases. In some implementations, the suggestedbid amount may be sent to a third-party content provider as a suggestedbid adjustment value. Such an adjustment value may be a suggestedmodification to an existing bid amount for one type of device, to entera bid amount for another type of device as part of the same campaign.For example, content management system 108 may suggest that athird-party content provider increase or decrease their bid amount fordesktop devices by a certain percentage, to create a bid amount formobile devices.

Analysis system 150 may interact with user devices 104, content providerdevices 106, content management system 108, and/or various other devicesand/or systems to collect data for use in performing analysis of organicsearch keywords in conversion path data and generating and providingrecommendations to content providers. Analysis system 150 may storeand/or retrieve data for use in performing various analyses in ananalysis database 160. As described in further detail with respect toFIGS. 2-5, according to illustrative implementations, analysis system150 may identify one or more organic search keywords 164 withinconversion path data 162 for a content provider. Analysis system 150 mayinclude an organic search keyword analysis module 152 configured toanalyze organic search keywords 164 and generate one or more analysismetrics 166 for each organic search keyword 164. For example, analysismetrics 166 may include a conversion contribution metric 168 related tohow directly the organic search keyword contributes to conversions(e.g., whether the keyword generally appears in connections withupper-funnel or lower-funnel events), a cost metric 170 indicating arelative cost to the content provider if the organic search keyword wereadded to the content provider's paid keywords 114, a new visitor metric172 related to how often within the conversion path data the organicsearch keyword is associated with driving new customers to a resource(e.g., website) of the content provider, and/or other types of metrics.In some implementations, analysis system 150 and content managementsystem 108 may be integrated within a single system (e.g., contentmanagement system 108 may be configured to incorporate some or all ofthe functions/capabilities of analysis system 150).

In some implementations, analysis system 150 may include a paid keywordrecommendation module 154. Paid keyword recommendation module 154 may beconfigured to generate recommendations to add one or more of organicsearch keywords 164 to the content provider's paid keywords 114 based onanalysis metrics 166 generated by organic search keyword analysis module152. In some implementations, analysis system 150 may additionally oralternatively include a new content recommendation module 156 configuredto generate recommendations for types of new content the contentpublisher may wish to consider generating and/or publishing (e.g., onthe publisher's website) based on selected organic search keywords 164determined based on analysis metrics 166. Modules of analysis system150, such as modules 152, 154, and/or 156, may be implemented asinstructions stored within a computer-readable storage medium operablycoupled to analysis system 150 and executable by at least one processorof analysis system 150.

FIG. 2 illustrates a flow diagram of a process 200 for generatingrecommendations for paid keywords based on analysis of organic searchkeywords in conversion path data according to an illustrativeimplementation. Referring to both FIGS. 1 and 2, analysis system 150 maybe configured to determine conversion path data 162 for a contentprovider (205). Conversion path data 162 includes data relating tomultiple conversion paths associated with the content provider leadingto multiple conversions. A conversion may include a sale of a product orservice, receipt of one or more information items from a user, receiptof a communication (e.g., a phone call) from a user, and/or any othertype of user activity that the content provider desires for users toperform and/or that represents some value to the content provider. Insome implementations, each conversion path may begin with one or moreuser interactions and end with a converting activity. The interactionsmay be instances where impressions of a content item have been displayedon the user device of the user, instances where the user clicked throughor otherwise selected the content item, instances where the userconverted, etc. In various implementations, conversion path data 162 maybe generated by analysis system 150 and/or received from one or moreexternal systems.

Conversion path data 162 may include paid interactions, or interactionsassociated with content items that were selected by content managementsystem 108 (e.g., through an auction process) in response to bidssubmitted by the content provider in association with paid keywords 114.Conversion path 162 may also include organic interactions, orinteractions associated with content items, such as search results, thatwere not provided to the user as a result of paid bids and associatedpaid keywords 114. For example, when a user submits a search querythrough a search interface, the user may be provided with a set of paidcontent items (e.g., text items, images, etc.) and a set of organiccontent items, such as links to resources (e.g., websites) deemedrelevant to the search query by the search engine. In this example, whenconversion path data 162 reflects user selection of an organic contentitem, some or all of the keywords of the search query leading toselection of the content item may be considered organic search keywords(e.g., if the keywords are not a part of paid keywords 114). In someimplementations, some or all interaction instances within conversionpath data 162 may include one or more keywords associated with theinteraction.

FIG. 3A provides a visual representation of a basic set of conversionpath data 350 according to an illustrative implementation. Conversionpath data 350 is associated with a fictional Acme Shoe Company.Conversion path data 350 includes a first conversion path 360 leading toa first conversion 370, and a second conversion path 380 leading to asecond conversion 390. Each of conversion paths 360 and 380 includesmultiple nodes associated with user interactions that ultimately lead toconversions 370 or 390. Nodes near the top of FIG. 3A representupper-funnel interactions, or interactions that occur near the beginningof the conversion paths, while nodes near the bottom representlower-funnel interactions, or interactions that occur near the endconversions 370 and 390.

In the illustrated implementation, organic search interactions areillustrated using circular shapes, and non-organic, or paid,interactions are illustrated using rectangular shapes. A very simpleconversion path data set has been illustrated here for the purposes ofclarity; it should be understood that a typical conversion path data setwould include many more than two conversion paths and that theconversion paths may not necessarily be linear in nature. For example,in some conversion path data sets, some nodes may below to multipleconversion paths (i.e., some user interactions may contribute tomultiple conversions). Additionally, the conversion path data need notbe associated with a single source (e.g., a single search engine); insome implementations, the conversion path data may be aggregated fromvarious different sources.

Conversion path 360 begins with an organic interaction 362. Interaction362 may represent user selection of a content item, such as a searchresult, associated with “Acme URL 1” in a search results interface. Thesearch result associated with “Acme URL 1” is displayed in the searchresults interface in response to the user entering a search query in asearch engine interface including the keyword “Marathon Training.” Thesearch result is displayed within a set of organic search results in thesearch results interface and is not displayed as the result of a paidbid submitted by Acme.

The second interaction in conversion path 360 is a non-organicinteraction 364. Interaction 364 may represent user selection of acontent item (e.g., search result, display image, etc.) “Acme Boot Item1” that is displayed as the result of a paid bid submitted Acme todisplay the content item. The content item is displayed in response touser submission of a search query including the keyword “Running Shoes.”

Conversion path 360 also includes another organic interaction 366 aswell as another non-organic interaction 368. After non-organicinteraction 368, a converting activity 370 occurs, as indicated by thetriangular shape. In the illustrated implementation, the convertingactivity is a purchase of an Acme Cross-Trainers shoe product by theuser on Acme's website.

Conversion path 380 also begins with an organic interaction 382 in whichthe user is presented with a content item (e.g., search result)associated with “Acme URL 3” in the search results interface in responseto a search query including the keyword “Industrial Safety.” Conversionpath 380 includes two subsequent non-organic interactions 384 and 386that are displayed as a result of paid bids submitted by Acme. A finalorganic interaction 388 resulting from a search query including thekeyword “Comfortable Steel Toe Boot” leads to conversion 390, in whichthe user purchases an Acme Boot product on Acme's website.

Analysis system 150 may store conversion path data such as data 350visually represented in FIG. 3A in a database or other memory structure.The conversion path data may be stored in a lookup table, linked list,matrix, or any other data structure capable of preserving theinteraction data reflected within the conversion paths, including anindicator of the content items selected, the keywords associated withthe interactions, and the relationships between interactions (e.g.,where in the conversion path the interactions occurred). In someimplementations, the data associated with each node in the conversionpath may be stored within a single data structure or set of datastructures. In some implementations, the data may be stored in separatedata structures and/or locations and referenced to one another by anidentifier (e.g., a node identifier). For example, in someimplementations, all keywords in the conversion path data may be storedin one data structure, and all content identifiers may be stored in adifferent data structure.

Referring again to FIGS. 1 and 2, analysis system 150 may be configuredto determine organic search keywords 164 included within conversion pathdata 162 (210). In some implementations, analysis system 150 may isolateorganic search keywords 164 within conversion path data 162. Forexample, analysis system 150 may remove paid keywords 114 from a set ofall keywords reflected in conversion path data 162 stored in database160, such that the only keywords left in the set are organic searchkeywords 164. In some implementations, conversion path data 162 may beconfigured such that it identifies which keywords and/or interactionsare associated with organic activity and which are associated with paidactivity.

In the illustrative conversion path data 350 shown in FIG. 3A, analysissystem 150 may determine the set of organic search keywords by includingonly those keywords associated with interactions 362, 366, 382, and 388in the set of organic search keywords, as these nodes are identified asbeing associated with organic interactions. If the conversion path datadoes not differentiate between organic and non-organic interactions,analysis system 150 may determine the organic search keywords by takinga set of all keywords associated with notes in the conversion path dataand removing any keywords appearing in paid keywords 114.

Analysis system 150 may perform analysis (e.g., automated analysis) onorganic search keywords 164 to generate one or more analysis metrics 166to be used in generating recommendations (215). In some implementations,analysis system 150 may calculate a conversion contribution metric 168for each of organic search keywords 164 related to how directly theorganic search keyword contributes to the conversions reflected inconversion path data 162. For example, conversion contribution metric168 may provide an indication of whether the organic keyword tends toappear near the top of the marketing funnel reflected in conversion pathdata 162 (e.g., in early user interactions, further away from theconversions, such as near the top of the visual representation shown inFIG. 3A), or whether the organic keyword tends to appear near the bottomof the marketing funnel (e.g., in later user interactions, closer to theconversions, such as near the bottom of the visual representation shownin FIG. 3A).

In some implementations, conversion contribution metric 168 may includean assist-to-last ratio. The assist-to-last ratio may be implemented asa measure of the number of conversion paths that include the keywordversus the number of times the keyword appears as the last click beforeconversion in a path. Thus, in such an implementation, an assist-to-lastratio of one indicates that the keyword is a last-click keyword in everyconversion path. An assist-to-last ratio substantially lower than oneindicates that the keyword appears in significantly more conversionpaths as a last-click keyword than an assisting keyword, and mayindicate that the keyword is generally a lower-funnel keyword (generallyappears lower, or nearer the end conversions, in the conversion paths).An assist-to-last ratio substantially higher than one indicates that thekeyword appears in significantly more conversion paths as an assistingkeyword than a last-click keyword, and may indicate that the keyword isgenerally a higher-funnel keyword (generally appears higher, or furtheraway from the end conversions, in the conversion paths). In someimplementations, the assist-to-last ratio may be implemented as ameasure of the number of times a keyword appears in the conversion pathdata as an assisting keyword versus the number of times the keywordappears as the last click before conversion in the path. In someimplementations, the assist-to-last ratio may be anassist-click-to-last-click ratio (e.g., a number of assistclicks/selections, or clicks associated with the keyword that were notthe last click prior to conversion, versus a number of clicks associatedwith the keyword that were the last click prior to conversion). In someimplementations, the assist-to-last ratio may be aclick-assisted-conversions-to-last-click-conversions ratio (e.g., anumber of conversion paths on which the keyword was associated with anassist click versus a number of conversion paths on which the keywordwas associated with the last click before conversion). In someimplementations, the assist-to-last ratio may be anassist-impressions-to-last-click ratio (e.g., a number of assistimpressions, or impressions associated with the keyword that were notthe last impression shown prior to conversion, versus a number of clicksassociated with the keyword that were the last click prior toconversion). In some implementations, the assist-to-last ratio may be animpression-assisted-conversions-to-last-click-conversions ratio (e.g., anumber of conversion paths on which the keyword was associated with anassist impression versus a number of conversion paths on which thekeyword was associated with the last click before conversion). Invarious other implementations, other types of conversion contributionmetrics (e.g., a first-to-last ratio relating to a number of times thekeyword appears as a first click versus the number of times it appearsas a last click, an average position metric indicating the averageposition in which the keyword appears in the conversion paths, etc.) maybe calculated and used in generating recommendations.

In some implementations, analysis system 150 may additionally oralternatively generate other analysis metrics 166. For example, analysissystem 150 may generate a cost metric 170 indicating a relative cost tothe content provider if the content provider adds the organic searchkeyword to paid keywords 114. Cost metric 170 may be an estimatedabsolute cost (e.g., an estimated monetary cost to the content providerover a particular time period) or a relative cost (e.g., a cost comparedto current paid keywords 114, such as on a scale of one to ten).Analysis system 150 may be configured to generate cost metric 170 basedon previous bid data for the organic search keyword (e.g., bids placedby other content providers on the keyword). In some implementations, thedata used to generate cost metric 170 may be received from contentmanagement system 108.

In some implementations, analysis system 150 may generate a new visitormetric 172. New visitor metric 172 may be related to how often withinconversion path data 162 the organic search keyword is associated withdriving new customers of the content provider to a resource (e.g., awebsite) of the content provider and/or to a conversion. Analysis system150 may be configured to determine whether interactions reflected inconversion path data 162 are associated with new or returning customersbased on user device identifiers (e.g., browser application identifiers,such as browser cookies) and/or based on customer data received from thecontent provider. In some implementations, new visitor metric 172 may bea number of interactions associated with the organic search keywordwhere the user was a potential new customer, a ratio of interactionsassociated with a potential new customer versus interactions associatedwith a previous customer, or another type of metric.

In some implementations, analysis system 150 may generate a scale metricrelating to a frequency with which the organic search keyword appearedwithin conversion path data 162. The scale metric may include a numberof times the organic keyword appeared in conversion path data 162, apercentage or ratio associated with the keywords (e.g., versus the totalnumber of organic keywords, total number of paid keywords, total numberof combined keywords, etc.), or some other metric representative ofand/or related to the prominence of the organic search keyword withinconversion path data 162.

In various other implementations, analysis system 150 may generate andutilize various other metrics or secondary factors in generatingrecommendations. For example, analysis system 150 may generaterecommendations based on a device type identifier (e.g., mobile, tablet,desktop, etc.) and/or a location identifier (e.g., country, state, city,region, cellular tower area, etc.). In some implementations, contentproviders may provide one or more custom metrics for use in generatingrecommendations, such as an estimated customer lifetime value, a desiredprofit margin, to achieve a desired impact using newly recommendedkeywords.

Analysis system 150 may use analysis metrics 166 to generate one or morekeyword recommendations 174 for new keywords to be added to the contentprovider's paid keywords 114 based on the analysis metrics for organicsearch keywords 164 (220). In some implementations, analysis system 150may select the organic search keywords upon which keywordrecommendations 174 will be based by comparing analysis metrics 166 toone or more thresholds. In some such implementations, any organic searchkeywords having an analysis metric above an upper threshold or below alower threshold may be selected as the basis for keyword recommendations174. In some implementations, a predetermined number of organic searchkeywords having highest or lowest analysis metrics may be selected asthe basis for keyword recommendations 174. In some implementations, theanalysis metrics upon which keyword recommendations 174 are based may bea single type of analysis metric (e.g., conversion contribution metric168) or a combination of multiple types of metrics (e.g., a weightedcombination of conversion contribution metric 168, cost metric 170, newvisitor metric 172, a scale metric, and/or other metrics). In someimplementations, the metrics used to determine keyword recommendations174 and/or weights applied to the metrics in determining therecommendations may be at least partially customizable by the contentprovider.

In some implementations, keyword recommendations 174 may be generated bycomparing an assist-to-last ratio for each of organic search keywords164 against one or more threshold assist-to-last ratio levels. Forexample, an assist-to-last ratio for each organic keyword may becompared to a lower assist-to-last threshold (e.g., lower than 1, suchas 0.2). If the assist-to-last ratio for an organic keyword is below thelower threshold, this may indicate that the organic keyword is generallya lower-funnel keyword that tends to be more directly associated withconversions (e.g., generally appears within the last severalinteractions prior to conversion). In some implementations, suchkeywords may be included within keyword recommendations 174 as keywordsthat may be added to help directly drive additional conversions. In someimplementations, the assist-to-last ratio for each organic searchkeyword may be compared to an upper assist-to-last ratio (e.g., above 1,such as 10). If the assist-to-last ratio for an organic keyword is abovethe upper threshold, this may indicate that the organic keyword isgenerally an upper-funnel keyword associated with user interactionsearly in the conversion paths (e.g., generally appears within the firstseveral interactions). In some implementations, such keywords may beincluded within keyword recommendations 174 as keywords that may beadded to help raise brand awareness for the content provider when usersare early in their search processes, which may also ultimately helpdrive later conversions.

In some implementations, keyword recommendations 174 may includerecommendations to adopt selected organic search keywords includedwithin conversion path data 162 within paid keywords 114. In someimplementations, keyword recommendations 174 may additionally oralternatively include recommendations to adopt new keywords based onorganic search keywords within conversion path data 162 that may bedifferent from the organic search keywords themselves. For example, oneor more organic search keywords selected based on analysis metrics 166may be processed (e.g., through an adjacency engine) to determineadditional keywords related to the processed organic search keywords.For example, an organic search keyword “shoe” may be processed throughan adjacency engine to identify related keywords such as “boot,”“sneaker,” etc. In some implementations, keyword recommendations 174 maybe generated that recommend adoption of such adjacent keywords as partof paid keywords 114.

In some implementations, analysis system 150 may be configured toreceive input from the content provider and use the input to generatekeyword recommendations 174. For example, a content provider may providegoal information indicating that the content provider wishes to increaseawareness of its brand, more directly drive additional conversions,identify possible new paid keywords that may not be the focus ofcompetitors and may be less expensive than other, more popular keywords,etc. Analysis system 150 may be configured to account for the goal ingeneration keyword recommendations 174. For example, if the goal of thecontent provider is to increase early awareness of its brand, analysissystem 150 may place an emphasis on upper-funnel organic search keywordswhen generating recommendations 174. In some implementations, analysissystem 150 may be configured to allow the content provider to provideinput on or directly set one or more thresholds used in generatingrecommendations, such as an assist-to-last threshold.

Analysis system 150 may be configured to provide the generated keywordrecommendations 174 to the content provider (225). Analysis system 150may transmit keyword recommendations 174 to content provider devices 106through network 102. In some implementations, the recommendations may beprovided within a network-accessible user interface accessed usingcontent provider devices 106, an application resident on contentprovider devices 106, a report transmitted to the content providers(e.g., to an email account of the content providers accessible bycontent provider devices 106), etc.

In some implementations, analysis system 150 may be configured toreceive a selection of one or more of keyword recommendations 174 by thecontent provider (230). For example, analysis system 150 may provide anoption to the content providers to accept and/or reject one or more ofkeyword recommendations 174. Analysis system 150 may be configured toadd new keywords associated with the selected recommendations to paidkeywords 114 (235). In some implementations, analysis system 150 may addthe keywords automatically without requiring further intervention fromthe content provider. In some implementations, analysis system 150 mayrequest that the content provider select one or more options associatedwith a new keyword bid after receiving the recommendation selection.

In some implementations, analysis system may be configured to use thereaction of the content provider (e.g., the acceptance or rejection) todetermine one or more subsequent recommendations to provide to thecontent provider. For example, if the content provider rejects asuggestion to add the organic search keyword “marathon training” to paidkeywords 114, analysis system 150 may be configured to exclude similarorganic keywords, such as “marathon running” or “long-distance runtraining,” from being recommended in future recommendations. In anotherexample, if the content provider accepts a suggestion to add “marathontraining” to paid keywords 114, analysis system 150 may increase alikelihood of providing recommendations of similar types of organickeywords in similar recommendations. In various implementations,adjustments based on the content provider's reactions may be implementedas exclusion lists, weights applied to emphasize or deemphasize similarorganic keywords, and/or in another manner.

FIG. 3B illustrates a user interface 300 in which keywordrecommendations 174 may be provided according to an illustrativeimplementation. Interface 300 illustrates keyword recommendationsprovided for the fictional Acme Shoe Company. An upper-funnelrecommendation section 305 provides recommendations that typicallyappear near the top of the marketing funnel in conversion path data 162and may help increase early awareness of Acme's brand. A lower-funnelrecommendation section 310 provides recommendations that typicallyappear near the bottom of the marketing funnel in conversion path data162 and may help more directly drive additional conversion. In theillustrative implementation shown in FIG. 3B, the content provider isprovided with the organic search keywords associated with eachrecommendation, an indication of conversion contribution 168, and anindication of cost data 170. In the illustrated implementation, theconversion contribution data and cost data are provide on a relativescale of 1 to 10 (e.g., in comparison with other data in conversion pathdata 162 and/or other keywords, such as in paid keywords 114). In someimplementations, raw or absolute data may be provided rather thanrelative data. In some implementations, different, less, or additionaldata may be shown in interface 300, such as an indication of howfrequently the organic keywords were associated with actions bypotential new customers of the content provider, how frequently theorganic keywords appeared in conversion path data 162, etc. Interface300 also includes an accept button 315 and reject button 320 that may beused by the content provider to accept each recommendation forimplementation in paid keywords 114 or reject each recommendation andremove it from the list of recommendations.

In some implementations, analysis system 150 may be configured toevaluate the impact of newly added keywords to the performance of paidkeywords 114 after the keywords are added to paid keywords 114. Forexample, an organic keyword may be suggested by analysis system 150,adopted by the content provider, added to paid keywords 114, and lead tofew or no appreciable incremental conversions (e.g., a “false positive”opportunity). In such an example, analysis system 150 may be configuredto identify the fact that the addition of the organic keyword has nothad a substantial positive effect on the performance of paid keywords114, and may suggest discontinuing use of the organic keyword withinpaid keywords 114. In some implementations, the effectiveness of a neworganic keyword may be validated through controlled experimentation thatlooks at improved performance, for example, in overall conversiontotals, cost per acquisition (CPA) for the content provider, etc. Insome implementations, analysis system 150 may evaluate increased assistvolume for the new keyword in the converting paths in newly generatedconversion path data after the keyword is added to paid keywords 114.Increased, or in this case new, investment in a keyword should generallyresult in the keyword being present in more converting customer paths.In some implementations, auction position and/or click volume may beused by analysis system 150 as an indicator of whether the investment inthe new keyword is driving measurable increases in volume. Using volumeof paid and organic search traffic for the new keyword may also helpestablish if the investment was effective at cannibalizing from freetraffic.

In some implementations, analysis system 150 may additionally oralternatively be configured to generate recommendations for new contentthat a content provider may wish to publish based on analysis of organicsearch keywords 164. Referring now to both FIGS. 1 and 4, a flow diagramof a process 400 for generating new content recommendations based onanalysis of organic search keywords in conversion path data is shownaccording to an illustrative implementation. Under process 400, analysissystem 150 may determine conversion path data 162 (205), determineorganic search keywords 164 within conversion path data 162 (210), andanalyze organic search keywords 164 to generate one or more analysismetrics 166 (215) in a manner similar to that described above withrespect to process 200 for generating keyword recommendations 174.

Analysis system 150 may be configured to select one or more of organicsearch keywords 164 based on analysis metrics 166 (420). In someimplementations, analysis system 150 may be configured to select theorganic keywords by comparing the analysis metric for each keyword to athreshold metric. For example, analysis system 150 may compare anassist-to-last ratio for each of organic search keywords 164 to an upperthreshold assist-to-last ratio (e.g., to identify organic searchkeywords frequently utilized early in users' search processes) and/or alower threshold assist-to-last ratio (e.g., to identify organic searchkeywords frequently utilized close to when conversions occur). In someimplementations, analysis system 150 may additionally or alternativelyselect organic search keywords by selecting a predetermined number oforganic keywords having a highest or lowest analysis metric or by usinganother method, such as those described above with respect to FIG. 2.

Analysis system 150 may generate one or more new content recommendations176 based on the selected organic search keywords (425). In someimplementations, analysis system 150 may provide the organic keywordsthemselves to content providers and suggest developing content (e.g.,webpages, mobile applications, etc.) related to the subject matter ofthe organic keywords. In some implementations, analysis system 150 mayexpand upon the organic keywords to develop a more completerecommendation, either automatically or manually through an operator ofanalysis system 150 (e.g., a sales representative). Contentrecommendations 176 may be provided to a content provider through acontent recommendation interface presented on content provider devices106. In some implementations, a content provider may be able to providea reaction (e.g., approval or rejection) to content recommendations 176through the interface. The reaction may be used in developing subsequentcontent recommendations 176.

FIG. 5 illustrates a user interface 500 through which contentrecommendations 176 may be provided according to an illustrativeimplementation. Interface 500 includes a list of organic keywords and anindication related to conversion contribution 168, on a scale of 1 to10, with 1 representing a highest position in the marketing funnel and10 representing a lowest position. In some implementations, organickeywords may only be included in interface 500 if a scale metric for theorganic keywords indicates they appear at least a threshold number orpercentage within conversion path data 162. In some implementations, anindication of a scale metric may be provided within interface 500.

In some implementations, analysis system 150 may include a new contentanalysis module 158 configured to evaluate a contribution of new orexisting content published by the content provider and/or makerecommendations based on the evaluation. In some implementations, afterthe content provider adds a new content item (e.g., new webpage), newcontent analysis module 158 may analyze organic search activity leadingto traffic to the new content, as reflected within new conversion pathdata including activity leading to the new content. New content analysismodule 158 may make recommendations for further content the contentprovider may consider adding based on the analysis of the traffic and/ororganic search keywords leading to the new content. New content analysismodule 158 may additionally or alternatively suggest one or more newpaid keywords that the content provider may wish to add to paid keywords114 to scale the volume of traffic to the new content. In someimplementations, the new content recommendations and/or new paid keywordrecommendations may be generated based on an analysis of the organicsearch keywords leading to the new resource in a manner similar to thatdescribed above with respect to processes 200 and/or 400.

FIG. 6 illustrates a depiction of a computer system 600 that can beused, for example, to implement an illustrative user device 104, anillustrative content management system 108, an illustrative contentprovider device 106, an illustrative analysis system 150, and/or variousother illustrative systems described in the present disclosure. Thecomputing system 600 includes a bus 605 or other communication componentfor communicating information and a processor 610 coupled to the bus 605for processing information. The computing system 600 also includes mainmemory 615, such as a random access memory (RAM) or other dynamicstorage device, coupled to the bus 605 for storing information, andinstructions to be executed by the processor 610. Main memory 615 canalso be used for storing position information, temporary variables, orother intermediate information during execution of instructions by theprocessor 610. The computing system 600 may further include a read onlymemory (ROM) 610 or other static storage device coupled to the bus 605for storing static information and instructions for the processor 610. Astorage device 625, such as a solid state device, magnetic disk oroptical disk, is coupled to the bus 605 for persistently storinginformation and instructions.

The computing system 600 may be coupled via the bus 605 to a display635, such as a liquid crystal display, or active matrix display, fordisplaying information to a user. An input device 630, such as akeyboard including alphanumeric and other keys, may be coupled to thebus 605 for communicating information, and command selections to theprocessor 610. In another implementation, the input device 630 has atouch screen display 635. The input device 630 can include a cursorcontrol, such as a mouse, a trackball, or cursor direction keys, forcommunicating direction information and command selections to theprocessor 610 and for controlling cursor movement on the display 635.

In some implementations, the computing system 600 may include acommunications adapter 640, such as a networking adapter. Communicationsadapter 640 may be coupled to bus 605 and may be configured to enablecommunications with a computing or communications network 645 and/orother computing systems. In various illustrative implementations, anytype of networking configuration may be achieved using communicationsadapter 640, such as wired (e.g., via Ethernet), wireless (e.g., viaWiFi, Bluetooth, etc.), pre-configured, ad-hoc, LAN, WAN, etc.

According to various implementations, the processes that effectuateillustrative implementations that are described herein can be achievedby the computing system 600 in response to the processor 610 executingan arrangement of instructions contained in main memory 615. Suchinstructions can be read into main memory 615 from anothercomputer-readable medium, such as the storage device 625. Execution ofthe arrangement of instructions contained in main memory 615 causes thecomputing system 600 to perform the illustrative processes describedherein. One or more processors in a multi-processing arrangement mayalso be employed to execute the instructions contained in main memory615. In alternative implementations, hard-wired circuitry may be used inplace of or in combination with software instructions to implementillustrative implementations. Thus, implementations are not limited toany specific combination of hardware circuitry and software.

Although an example processing system has been described in FIG. 6,implementations of the subject matter and the functional operationsdescribed in this specification can be carried out using other types ofdigital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.

Implementations of the subject matter and the operations described inthis specification can be carried out using digital electroniccircuitry, or in computer software embodied on a tangible medium,firmware, or hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Implementations of the subject matter described inthis specification can be implemented as one or more computer programs,i.e., one or more modules of computer program instructions, encoded onone or more computer storage medium for execution by, or to control theoperation of, data processing apparatus. Alternatively or in addition,the program instructions can be encoded on an artificially-generatedpropagated signal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. A computer storage medium can be, or be includedin, a computer-readable storage device, a computer-readable storagesubstrate, a random or serial access memory array or device, or acombination of one or more of them. Moreover, while a computer storagemedium is not a propagated signal, a computer storage medium can be asource or destination of computer program instructions encoded in anartificially-generated propagated signal. The computer storage mediumcan also be, or be included in, one or more separate components or media(e.g., multiple CDs, disks, or other storage devices). Accordingly, thecomputer storage medium is both tangible and non-transitory.

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” or “computing device” encompassesall kinds of apparatus, devices, and machines for processing data,including by way of example, a programmable processor, a computer, asystem on a chip, or multiple ones, or combinations of the foregoing.The apparatus can include special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application-specificintegrated circuit). The apparatus can also include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, across-platform runtime environment, a virtual machine, or a combinationof one or more of them. The apparatus and execution environment canrealize various different computing model infrastructures, such as webservices, distributed computing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example, semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be carried out using acomputer having a display device, e.g., a CRT (cathode ray tube) or LCD(liquid crystal display) monitor, for displaying information to the userand a keyboard and a pointing device, e.g., a mouse or a trackball, bywhich the user can provide input to the computer. Other kinds of devicescan be used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Implementations of the subject matter described in this specificationcan be carried out using a computing system that includes a back-endcomponent, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front-endcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described in this specification, or anycombination of one or more such backend, middleware, or frontendcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), an inter-network (e.g., theInternet), and peer-to-peer networks (e.g., ad hoc peer-to-peernetworks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someimplementations, a server transmits data (e.g., an HTML page) to aclient device (e.g., for purposes of displaying data to and receivinguser input from a user interacting with the client device). Datagenerated at the client device (e.g., a result of the user interaction)can be received from the client device at the server.

In some illustrative implementations, the features disclosed herein maybe implemented on a smart television module (or connected televisionmodule, hybrid television module, etc.), which may include a processingcircuit configured to integrate internet connectivity with moretraditional television programming sources (e.g., received via cable,satellite, over-the-air, or other signals). The smart television modulemay be physically incorporated into a television set or may include aseparate device such as a set-top box, Blu-ray or other digital mediaplayer, game console, hotel television system, and other companiondevice. A smart television module may be configured to allow viewers tosearch and find videos, movies, photos and other content on the web, ona local cable TV channel, on a satellite TV channel, or stored on alocal hard drive. A set-top box (STB) or set-top unit (STU) may includean information appliance device that may contain a tuner and connect toa television set and an external source of signal, turning the signalinto content which is then displayed on the television screen or otherdisplay device. A smart television module may be configured to provide ahome screen or top level screen including icons for a plurality ofdifferent applications, such as a web browser and a plurality ofstreaming media services (e.g., Netflix, Vudu, Hulu, etc.), a connectedcable or satellite media source, other web “channels”, etc. The smarttelevision module may further be configured to provide an electronicprogramming guide to the user. A companion application to the smarttelevision module may be operable on a mobile computing device toprovide additional information about available programs to a user, toallow the user to control the smart television module, etc. In alternateimplementations, the features may be implemented on a laptop computer orother personal computer, a smartphone, other mobile phone, handheldcomputer, a tablet PC, or other computing device.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular implementations of particularinventions. Certain features that are described in this specification inthe context of separate implementations can also be carried out incombination or in a single implementation. Conversely, various featuresthat are described in the context of a single implementation can also becarried out in multiple implementations, separately, or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can, in some cases, beexcised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.Additionally, features described with respect to particular headings maybe utilized with respect to and/or in combination with illustrativeimplementations described under other headings; headings, whereprovided, are included solely for the purpose of readability and shouldnot be construed as limiting any features provided with respect to suchheadings.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products embodied on tangible media.

Thus, particular implementations of the subject matter have beendescribed. Other implementations are within the scope of the followingclaims. In some cases, the actions recited in the claims can beperformed in a different order and still achieve desirable results. Inaddition, the processes depicted in the accompanying figures do notnecessarily require the particular order shown, or sequential order, toachieve desirable results. In certain implementations, multitasking andparallel processing may be advantageous.

What is claimed is:
 1. A method comprising: determining, at acomputerized analysis system, conversion path data for a contentprovider, wherein the conversion path data comprises data relating to aplurality of conversion paths associated with the content providerleading to a plurality of conversions, and wherein each of the pluralityof conversion paths comprises one or more user actions leading to one ofthe plurality of conversions, the conversion path data comprising paidkeywords associated with bids submitted by the content provider andorganic search keywords that are not associated with paid bids by thecontent provider; filtering, at the analysis system, the conversion pathdata by removing the paid keywords from the conversion path data, anddetermining, based on the filtered conversion path data, a plurality oforganic search keywords; analyzing, at the analysis system, theplurality of organic search keywords within the conversion path data togenerate an assist-to-last ratio for each of the plurality of determinedorganic search keywords, the assist-to-last ratio comprising a measureof a first number of times the organic search keyword is an assistingkeyword not associated with a last selection of a content item prior toone of the plurality of conversions versus a second number of times theorganic search keyword is a last-click keyword associated with the lastselection prior to one of the plurality of conversions; determining, foreach organic search keyword, a relative position within the conversionpaths associated with the organic search keyword using theassist-to-last ratio for the organic search keyword; generating arecommendation and causing a display device to display therecommendation, the recommendation comprising an indication of therelative position of one or more of the plurality of organic searchkeywords, wherein the recommendation prompts the content provider toaccept or reject, via the display device, each of the one or moreorganic search keywords; and receiving an indication of accepted organicsearch keywords of the one or more of the organic search keywords fromthe display device, the accepted organic keywords accepted by thecontent provider; and adding the accepted organic search keywords to thepaid keywords.
 2. The method of claim 1, the method further comprising:selecting one of the plurality of organic search keywords based on theassist-to-last ratio; and generating the recommendation to prompt thecontent provider to accept or reject adding the selected organic searchkeyword to the content provider's paid keywords; and wherein selectingone of the plurality of organic search keywords comprises, for each ofthe plurality of organic search keywords: comparing the assist-to-lastratio for the organic search keyword to a particular thresholdassist-to-last ratio; and determining whether to select the organicsearch keyword based on the comparison.
 3. The method of claim 2,further comprising: determining reaction data based on whether thecontent provider accepts or rejects the one or more organic searchkeywords; and determining one or more subsequent recommendations basedat least in part on the reaction data.
 4. The method of claim 2, furthercomprising providing a comparison between the selected organic searchkeyword of the recommendation and the paid keywords.
 5. The method ofclaim 2, further comprising evaluating a performance of the selectedorganic search keyword and generating a recommendation as to whether theselected organic search keyword should remain in the set of paidkeywords or be removed from the set of paid keywords based on theperformance.
 6. The method of claim 2, further comprising analyzing theplurality of organic search keywords within the conversion path data togenerate an analysis metric other than the assist-to-last ratio, andselecting one of the plurality of organic search keywords based on boththe analysis metric and the assist-to-last ratio.
 7. The method of claim6, wherein the analysis metric further comprises a cost metricindicating a relative cost to the content provider if the contentprovider adds the organic search keyword to the set of paid keywords. 8.The method of claim 6, wherein the assist-to-last ratio comprises ameasure of a first number of conversion paths on which the organicsearch keyword is an assisting keyword not associated with a lastselection prior to one of the plurality of conversions versus a secondnumber of conversion paths on which the organic search keyword is alast-click keyword associated with the last selection prior to one ofthe plurality of conversions.
 9. The method of claim 6, wherein theanalysis metric further comprises a scale metric indicating a frequencywith which the organic search keyword appears in the conversion pathdata.
 10. The method of claim 6, wherein the analysis metric furthercomprises a new visitor metric related to how often within theconversion path data the organic search keyword is associated withdriving new customers of the content provider to a resource of thecontent provider.
 11. A system comprising: at least one computing deviceoperably coupled to at least one memory and configured to: determineconversion path data for a content provider, wherein the conversion pathdata comprises data relating to a plurality of conversion pathsassociated with the content provider leading to a plurality ofconversions, and wherein each of the plurality of conversion pathscomprises one or more user actions leading to one of the plurality ofconversions, the conversion path data comprising paid keywordsassociated with bids submitted by the content provider and organicsearch keywords that are not associated with paid bids by the contentprovider; filter the conversion path data by removing the paid keywordsfrom the conversion path data and determine, based on the filteredconversion path data, a plurality of organic keywords; analyze theplurality of organic search keywords within the conversion path data togenerate an assist-to-last ratio for each of the plurality of organicsearch keywords, the assist-to-last ratio comprising a measure of afirst number of times the organic search keyword is an assisting keywordnot associated with a last selection of a content item prior to one ofthe plurality of conversions versus a second number of times the organicsearch keyword is a last-click keyword associated with the lastselection prior to one of the plurality of conversions; determine, foreach organic search keyword, a relative position within the conversionpaths associated with the organic search keyword using theassist-to-last ratio for the organic search keyword; generate arecommendation and cause a display device to display the recommendation,the recommendation comprising an indication of one or more funnelposition categories, each funnel position category comprising one ormore of the organic search keywords that include a relative positionassociated with the funnel position category, wherein the recommendationprompts the content provider to accept or reject, via the displaydevice, each of the one or more of the organic search keywords of theone or more funnel position categories; receive an indication of one ormore accepted organic search keyword of the one or more organic searchkeywords from the display device, the accepted organic search keywordsaccepted by the content provider; and add the accepted organic searchkeywords to the content provider's paid keywords.
 12. The system ofclaim 11, wherein the at least one computing device is configured to:select one of the plurality of organic search keywords by, for each ofthe plurality of organic search keywords: comparing the assist-to-lastratio for the organic search keyword to a particular thresholdassist-to-last ratio; and determining whether to select the organicsearch keyword based on the comparison; and generate the recommendation,the recommendation comprising an indication to add the selected organicsearch keyword to the content provider's paid keywords.
 13. The systemof claim 12, wherein the at least one computing device is configured toevaluate a performance of the selected organic search keyword andgenerate a recommendation as to whether the selected organic searchkeyword should remain in the set of paid keywords or be removed from thecontent provider's paid keywords based on the performance.
 14. Thesystem of claim 12, wherein the at least one computing device isconfigured to provide a comparison between the selected organic searchkeyword of the recommendation and the paid keywords.
 15. The system ofclaim 12, wherein the at least one computing device is configured toanalyze the plurality of organic search keywords within the conversionpath data to generate an analysis metric other than the assist-to-lastratio, and selecting one of the plurality of organic search keywordsbased on both the analysis metric and the assist-to-last ratio.
 16. Thesystem of claim 15, wherein the analysis metric further comprises ascale metric indicating a frequency with which the organic searchkeyword appears in the conversion path data.
 17. The system of claim 15,wherein the analysis metric further comprises a cost metric indicating arelative cost to the content provider if the content provider adds theorganic search keyword to the set of paid keywords.
 18. The system ofclaim 15, wherein the analysis metric further comprises a new visitormetric related to how often within the conversion path data the organicsearch keyword is associated with driving new customers of the contentprovider to a resource of the content provider.
 19. The system of claim15, wherein the conversion contribution metric comprises anassist-to-last ratio, wherein the assist-to-last ratio comprises ameasure of a first number of conversion paths on which the organicsearch keyword is an assisting keyword not associated with a lastselection prior to one of the plurality of conversions versus a secondnumber of conversion paths on which the organic search keyword is alast-click keyword associated with the last selection prior to one ofthe plurality of conversions.
 20. One or more non-transitorycomputer-readable storage media having instructions stored thereon that,when executed by at least one processor, cause the at least oneprocessor to perform operations comprising: determining conversion pathdata for a content provider, wherein the conversion path data comprisesdata relating to a plurality of conversion paths associated with thecontent provider leading to a plurality of conversions, and wherein eachof the plurality of conversion paths comprises one or more user actionsleading to one of the plurality of conversions, the conversion path datacomprising paid keywords associated with bids submitted by the contentprovider and organic search keywords that are not associated with paidbid by the content provider; filtering the conversion path data byremoving the paid keywords from the conversion path data anddetermining, based on the filtered conversion path data, a plurality oforganic keywords; analyzing the plurality of organic search keywordswithin the conversion path data to generate an analysis metric for eachof the plurality of organic search keywords, wherein the analysis metriccomprises a conversion contribution metric related to how directly theorganic search keyword contributes to the plurality of conversions,wherein the conversion contribution metric comprises an assist-to-lastratio which comprises a measure of a first number of times the organicsearch keyword is an assisting keyword not associated with a lastselection of a content item prior to one of the plurality of conversionsversus a second number of times the organic search keyword is alast-click keyword associated with the last selection prior to one ofthe plurality of conversions; determining, for each organic searchkeyword, a relative position within the conversion paths associated withthe organic search keyword using the assist-to-last ratio for theorganic search keyword; determining, for each organic search keyword,whether the organic search keyword is associated with one of a pluralityof funnel categories based on a comparison of the assist-to-last ratioof the organic search keyword to a plurality of assist-to-last ratiothresholds, wherein the plurality of funnel categories comprises anupper funnel category and a lower funnel category; generating arecommendation and causing a display device to display therecommendation, the recommendation comprising an indication of theplurality of funnel categories and the associations of one or more ofthe organic search keywords with the funnel categories, wherein the userinterface prompts the content provider to accept or reject each of theorganic search keywords; and receive an indication of one or moreaccepted organic search keyword of the organic search keywords from thedisplay device, the accepted organic search keywords accepted by thecontent provider, and add the accepted organic search keywords to thecontent provider's paid keywords.
 21. The one or more computer-readablestorage media of claim 20, wherein the analysis metric further comprisesat least one of a scale metric, a cost metric, and a new visitor metric,wherein the scale metric indicates a frequency with which the organicsearch keyword appears in the conversion path data, wherein the costmetric indicates a relative cost to the content provider if the contentprovider adds the organic search keyword to the set of paid keywords,and wherein the new visitor metric relates to how often within theconversion path data the organic search keyword is associated withdriving new customers of the content provider to a resource of thecontent provider.
 22. The one or more computer-readable storage media ofclaim 20, wherein the assist-to-last ratio comprises a measure of afirst number of conversion paths on which the organic search keyword isan assisting keyword not associated with a last selection prior to oneof the plurality of conversions versus a second number of conversionpaths on which the organic search keyword is a last-click keywordassociated with the last selection prior to one of the plurality ofconversions.
 23. The one or more computer-readable storage media ofclaim 22, the operations further comprising: selecting one of theplurality of organic search keywords based on the assist-to-last ratio;and generating a recommendation to add the selected organic searchkeyword to the content provider's paid keywords; wherein selecting oneof the plurality of organic search keywords comprises, for each of theplurality of organic search keywords: comparing the assist-to-last ratiofor the organic keyword to a particular threshold assist-to-last ratio;and determining whether to select the organic search keyword based onthe comparison.